ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

As mobile devices are becoming ubiquitous, regularly interacting with a variety of user interfaces (UIs) is common aspect daily life for many people. To improve the accessibility these and to enable their usage in settings, building models that can assist users accomplish tasks through UI vitally important. However, there several challenges achieve this. First, components similar appearance have different functionalities, making understanding function more important than just analyzing appearance. Second, domain-specific features like Document Object Model (DOM) web pages View Hierarchy (VH) applications provide signals about semantics elements, but not natural language format. Third, owing large diversity UIs absence standard DOM or VH representations, model high coverage requires amounts training data. Inspired by success pre-training based approaches NLP tackling problems data-efficient way, we introduce new pre-trained representation called ActionBert. Our methodology designed leverage visual, linguistic interaction traces pre-train generic feature representations components. key intuition actions, e.g., sequence clicks on components, reveals information functionality. We evaluate proposed wide downstream tasks, ranging from icon classification component retrieval its description. Experiments show ActionBert outperforms multi-modal baselines across all up 15.5%.